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ABSTBACT 

The overview serves as an introduction to seven 
evaluation reports produced for the Occupational Exploration Program 
(OEP) , a joint developaental effort of the Center for Tocational 
Education, Ohio State University, and the Jefferson Countj# Colorado 
public schools. The 1973-74 project vas designed to provide 
occupational exploration experiences for junior high school students* 
The report contains a description of evaluation procedures employed 
by the prograa as veil as an overall description of the program 
itself* The latter includes goals, objectives, and a discussion of 
simulation, the aajor technique used for delivering the program* The 
evaluation procedures section covers suaaative and formative 
evaluation concerns; general sampling; design parameters; and a brief 
description of the Reviser's Information Summary, the compilation of 
evaluation data that accompanies each of the seven OEP reports* The 
final section describes the evaluation instruments utilized and 
procedures folloved* Discussed are: knowledge tests, affective tests, 
student module questionnaires, teacher evaluation logs, panel review 
procedures, and observer forms* (Author/HH) 
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Evalui-.tion Repoz-t notebook - An O.'-ervicv 
Repoi-t rov the OccuT;at-l or. a?> Ex))loration Frogram 

By Jataec K. Altschuld 



This overview is to serve as cn introduction to seven evaluation reports 
produced for the Occupational Exploration Program. The Occupational Explo- 
ration Program (O.E.P.) is funded by the National Institute of Education 
(N.I.E.) and is a Joint developmental effort of the Center for Vocational 
Education (The Ohio State UniversitjO and the Jefferson Coun^, Colorado public 
schools . The overview-: contains a dtescription of the evaluation procedures 
employed for the program as well as an overall description of the program 
itself. The latter includes goals, objectives and a discussion of simulation, 
the ma^or technique used for delivering the program. The evaluation procedures 
section covers sumraative and formative evaluation concerns; general sampling 
and design parameters; instrumentation; and a brief description of the Revisor's 
Information SuKsaxy, a unique way of compiling information collected in the 
evaluation. 
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I. Introduction 

A. Purposes/Description of the Notebo ok 

The contents of the evaluation report notebook represent a compilation 
of the evaluation data collected during the pilot test of occupational exploration 
curriculum modules developed jointly by the Center for Vocational Education and 
the Jefferson County, Colorado Public Schools. The notebook is organized as 
follow;? : 

-niere is a ffont section in which overviews of the general 
nature of the project euxd the evaluation that was conducted 
are given. This introduction is part of that front material. 
-The next sections of the notebook contain specific module 
evaluation reports including a description of the actual 
evaluation that was in5)lemented for a module, data tables and 
interpretation, and of greatest importance, a "Revi »er^ 
Information Summary ". (Note: Module specific sections will 
contain outlines describing their individual content.) 
The notebook has been designed to serve several basic purposes. First, it 
will be a major record source for project personnel regarding pilot test data 
collected for the project in •FY' 73-7^ • The scope of the evaluation and the 
amount of data being collected necessitate that the data be summarized, 
collated and easily accessible for future use. Uie notebook will facilitate 
the process of accessing summarized data. Secondly, it will provide an over- 
view of the project evaluation for both project personnel as well as the project 
sponsor. Lastly, the Reviser *s Information Summary (RIS) provides the means for 
module revisers to start the revision process without having to wade through an 
excessive amount of data. If the reviser does desire to study data the notebook 
also includes data tables for his/her perused. 
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B, How to Use the Notebook 
1. For Revisers 

On initial use of the notebook it is recommended that the reviser 
first read the front section. This will enable him to understand 
the overall nature of the evaluation strategy engployed on the 
project. Next the reviser should review the module specific 
section which contains descriptions of how the evaluation was 
actually carried out, sairple used, type of test questions, etc., 
and skim Section III, Parts A, B, C and D. Section E probably 
should be read carefully. This review will help the reviser to 
understand the context in which the module was used and should 
help in interpreting the Reviser's Information Summary. Then he 
should proceed to the Reviser's Information Summary for the 
specific module and closely read the information provided. This 
coupled with his/her knowledge of the module should enable the 
reviser to begin to generate a revision strategy for the specific 
module. If the reviser feels the need to examine the data more 
closely, the next step would be to refer back to the appropriate 
data tables provided in the report. In other words ♦ after reading 
the RIS the reviser may use the data tables in accordance with his/ 
her individual information needs. When using the notebook for a 
second time the reviser most likely should omit reading this 
general front matter portion of the notebook. 
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2. For Evaluators^ Project Supervisory Personnel, NIE Staff 

As in the case of revisers , it is recdnnnended that evaluators, 
program supervisory personnel and NIE staff read the general 
front matter first and then read the entire module specific 
section. On second use of the notebook, it would then only 
be necesaary to read the module specific section. 

3- Speciel Note 

Throughout this front material, commentary, as already found In 
the text, has been added to assist the reviser in developing a 
perspective on the revision process* These comments are by 
design Just a sample of brief suggestions or things to consider 
as the reviser contemplates the revision process. 
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Project Description 

A. Overview had Tyves of Product 

The occupational exploration project was designed for the purpose 
CjT assisting students in the middle grades (8th and 9th grades) with the 
process of exploring occupations. It vas to not only help them explore occupa- 
tions , but also to aid them in developing fixture occupational and/or occupational 
exploration plans. The basic strategy for working vith students ^wls to produce 
curriculum materials that could be isiplemented vith small groups or classes of 
students. The Instructional technique for most of the materials produced in 
1973-7^ centered around slisulations that vould take approximately 3--^ ^eks 
of class time (1 hour per day). The sinulationa were built vith the foUoiring 
considerations in mind: 

-That they vould expose students to a variety of careers in 
the occupational area being simulated (Breadth of Coverage); 
-That they vould provide the students vith the opportunity to 
study at least one career in some depth (Depth of Coverage); 
-That they vould provide "hands on" or active types of roles 
for the students to play as opposed to a passive or a Celling" 
students type of approach (Hands On Experience); 
-That they vx^uld be heavily student directed, not self 
instructional but student directed* The teacher's role is 
more that of a facilitator, gulder or leader — a manager 
of instruction as opposed to a director of instruction 
(student Directed )i 
-That they vould include mi'^liated alternatives (slides, 
cassettes, films, talking pages, etc. ) at key points vithin 
the simulation so that stu'lents vould not alvsys *?ave ";o read 
the materials. 

io 



It vaa hoped that this approach vould enhance 
students* motivation as they worked their way through the 
materials (Mediated Motivation}* 
For the project year 1973-7**, 12 simulation modiaes were to he developed, 
with seven (7) of those modules to undergo extensive pilot testing in the 
Jefferson County, Colorado, and Denver, Colorado public schools* Where 
feasible, incremental testing (testing of module parts was to be carried 
out during the development of modules* The developmental, as weU as the 
testing schedule, is briefly outlined in Table I* 



Table I - Schedule of Development/Testing for Modules 



Time 

Autumn 1973 

Late Atxtumn, 1973«- 
Late Winter, 197^ 

Winter - Jlme 30, 197^ 



Development 
3 simulation modules 
k siMxlation modules 



5 simulation modules 
(Up to Pilot Test 
Readiness) 



Testing 
mot Test 



Pilot Test^ In- 
crementally Test 
Where Feasible 



Incrementally 
Test, Where 
Feasible 



A simulation uiodaile consists of four main parts* The first part, the 
Preview , consists of a short introductory section - usually a film or 
sound- slide presentation with an accompanying illustrated booklet. Its 
primary function is to motivate or stimulate student interest in participating 
in the simulation* In the ideal world, the individual student would elect to 
either participate or not participate after seeing/reading the preview. (As 
some schools are presently constituted, this may not be feasible and students 
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generally aight not be alloired to exercise the option )• After seeing/reading 
the preview, the students move into the Preparation Phase which enables them to 
Identify the role(8) they wlLl perform in the simulation. The students then 
enter into the activity or Participation Phase of the simulation. Here they 
perform various tasks which are related to their roles and help them to better 
unai:rstand the natixre of the role as it might occur in the World of Work. 
Upon completion of the participation phase, the students move Into the Suamaxy 
Phase of the module. In the sunmary phase, students analyze and pull to- 
gether their exploration experiences in the module. Hopefully, this will 
help them plan and prepare for future exploration experiences both of a formal, 
as well Qs an informal nature. 

The parts of a module are especially Important to the module reviser in- 
asmuch as the revision must deal with them specifically. In addition, th^^ 
reviser must pay attention to the transition points between the phases. 
Inadequate transitions may seriously effect the success of the module. Data 
win be summarized, to the extent possible, on a phase by phase basis to 
facilitate the revision process. The reviser should keep in mind that only 
a subset of potential revision questions could be answe*^ at at^one tiiut. 
Care therefore should be taken in reading the module and studying the revision 
information before the development of a revision strategy is undertaken. 

B. Goals/Objectives 

As stated earlier the basic purpose of the project was to help 
students explore careers and develop career exploration plans. More specie 
flcally a set of project goals and sub-goals were developed. These are 
listed below (Table II). The reader should keep in mind that while these 
goals {provided the focus for the development of modules, a much more specific 
set of goals - objectives is necessary for conducting the evaluation. As a 
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first step at establishing the objectives, CVE and Jeffco project staff took 
each sub-goal and delineated behaviors that might be representative of the goal. 
This effort resulted in an extensive listing of behaviors - one which vas fully 
beyond the present capability to evaluate. A further consideration was that 
some of the delineated behaviors, as well as the original goals, were more 
closely related to the overall Impact of a large scale program as opposed to 
th^ inteivention of one or two modules. Therefore, project staff decided upon 
a more basic approach to the interim assessment of module effects or intact - 
that is, a simplified set of behaviors was established for the pilot test of 
se/en (?) modules produced in 1973-7U. These behaviors, with the limitations 
given in the next paragraph were meCvBured ir the interim susnative evaluation 
of a nodule. 

This set of behaviors is listed in Table HI, Note that these behaviors 
were generated from an overall project point of view. A given module will 
not necessarily deliver equally on all of the behaviors and for that matter, 
a given module might deliver on only two or three of the set of behaviors. 
Specific module intents ai.d contents do differ within the general framework 
of the project. In each module specific section, a description of the be- 
haviors delivered by that module will be given. 
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Table II - Goals and Sub-Goals as Specified 
in the Project Proposal (1973-7^^) 

1#1 Sub-Goals Related to the Development of an Information Base 

IA0I To build an occupational information base regarding the 

environment, the people, and the processes associated with the 

world of work* 

1^1.2 To build a personal information base regarding interests, 
aptitudes, achievements, and values associated with the world of 
work^ 

1^1*3 To develop increased awareness and recognition of the ways 
in which people and occupations within a broad field of work 
and across several broad fields of work resemble or differ from 
one another « 

l^l.U To apply several occupational classification systems in 
characterizing broad fields of work. 



1*2 Sub-Goals Related to the Development of Attitudes 

1.2,1 To heighten one^s sensitivity tcjmrd perceiving and responding 
to experiences associated with the world of work. 
1*2*2 To develop personal feeD.ings and preferences toward 
occupational exploration experiences. 

1.2.3 To develop an appreciation for potential personal fulfilLnents 
associated with different occupational opportunities* 
1,3 Sub-Goals Related to the Development of Applicative Strategies 

1,3»1 To fonnulate and refine plans to explore in more relevant detail 
broad fields of work tased on a r>ynthesis of occupational and personal 
information ♦ 

1*3#2 To systematically apply exploration techniques to personally 



ERIC 14 



9 



appropriate occupational decisions* 

1.3-3 TO compare hypotheses regarding one's developing self and 
occupational preferences with one's past preferences and stages of 
development. 

1*3.** To generate and use a set of criteria to evaluate the 
qualitative realism and goodness of personally derived occupational 
exploration gocLLs. 



Table HI - Delineated Behaviors for Use 
in the Pilot Test (1973-7U) 

^REA TTPE OF BEHAVIOR 

Cognitive Skills Knowledge of: Processes 

Responsibilities 
Tools 

Environments 

Skills/Requirements 

Interactions 

Applications of the above areas 
where feasible 

Affective Skills Expressed llkes/dlsliKes (pre- 

ferences for) Jobs or functions 
in simulations as veil as reasons 
for same. 



C. Intended Audience and Use for the Simulations 

1. Audience - The target audience (for *Pi''^ Jk) modules is 8th and 
9th graders from so called "average" junior high schools. Average in this case 
is loosely defined as st hools which basically draw their populations from 
communities with a middle class socio-economic-status. Ttte materials as now 
constituted are not designed particularly for use in an inner city school* 

As for subject matter, the modules could flexibly be used in a variety of 
academic classrooms. Sevv^ral modules axe exceptions to the rule, but in 
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general it holds ♦ 

2. Use - The modxiles usually accommodate IO-15 students. Again 
the rule is general and several modules include roles for all the students 
in a class • There are four or five basic ways of obtaining or setting up 
groupings c f students for a module* These will be described under the 
sampling nart of this front section. 

in. Evaluation Procedures - Description of 
A. Overview 

A primary factor in designing the evalmtion strategy for the 
Occupational Exploration Program was that materials developed during the first 
year of the project would be in semi-finished but not final draft form. Hence, 
any testing or evaluation should focus primarily on the collection of revision 
data, i.e., the evaluati<»i should be formative in nature. Beyond the formative 
emphasis it seemed sensible to collect information regarding the interim effect/ 
ifl?)act of the materials on students. Therefore, a secondary enipha!:,is of the 
evaluation was a summative one. The distinction just made is to a degree 
arbitrary and semantic, but at the same time it serves a useiful purpose. That 
is, it provides a framework in which to describe the evaluation strategy. 

In addition to the above distinction the reader must note that the pilot 
test of each module was viewed as an individual experiment. A full exploration 
program will not be in operation until approximately 2-3 years from the date of 
project inception. The resultant effect is that in the cognitive and affective 
areas the assessment of only a limited set of objectives (as shown earlier in 
Table III) could be undertaken. 
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B. Types of Evaluation 

1. Formative Evaluation 

a. Purpc se 

The overall goal of formative evaluation f:>r the Occupational 
Exploration Program in 1973-7^ is to develop a bank of infor- 
mation relevant and useful for revision and refinement -*)f each 
module that is pilot tested. More specifically the formative 
evaluation will attempt to collect inf* rmatirn regarding: 
problems in implementing the modules; suggestions regaining 
the improvement of the module; and overall perceptions of the 
module. 

b. General Design and Information Sources 

Data for the formative evaluation was collected from two groups - 
teachers and students who participated in the module implementation. 
The basic approach was for teachers to complete sections of a 
questionnaire, the Teacher Evaluation Log, as they worked with and 
observed students progressing thr:^ugh a m dule. The questicnnaire 
was designed and ordered so that each point in it correlated with 
a specific part of the module. The last section of the Teacher 
Evaluation Log contained questions regarding the teacher's overall 
perception of the module. Besides completing the IDG, teachers 
also participated as a group in a post module debriefing session 
(Post f4odule Panel Review). Here they could freely* discuss what they 
considered to be the strengths and weaknesses of the module and the 



♦Note: The panel always did have a leader (a Jeff Co. Administrat' r) who helped 
tc keep the general discussion of a nK)dule on track and gear3d t' ward 
the eventual revision of a module. CVE and Jeff Co, personnel worked 
cl sely together tc establish guidelines for conducting the panels. 
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types of changes that they would make in it. The results of the 
panels vere recorded, and summarized for use in the revision process* 

For students, revision data was collected by means of 
questionnaires administered to students after they had completed 
a module and from pre- and posttests dealing with the afore- 
mentioned cognitive and affective behaviors. This latter set 
'f data is limited from a revision standpoint, however, for two 
reasons. First, the tests were developed from an overall 
module standpoint and it is difficult to pinpoint the data 
directly to a specific module section - a key concern for revisers. 
Secondly, and related to the first concern, is that the t-^sts were 
written for the purpose of assessing module effect /impact, not 
for diagnostic purposes. The reviser is, therefore, cautioned 
about placing undue emphasis upon the results from the 
cognitive and effective tests. 

There are several other minor sources of revision data. These 
include observations of Jeffco project staff, reviews of materials 
as they were developed, coranents of editors and product 
developers, etc. Where these are available, the revision 
information will be included on the Reviser *s Informtion Summary. 

In Table IV a brief overview of the formative evaluation 
portion of the pilot test is shown. 
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Table IV - Overview* of the Formative Evaluation Portion 
of the Pilot Test (1973-7^*) 



Information Source 



Timing Relative to Wnen 
Module is Taught 



1. Revisions 
Suggestions 




X 


1 
1 

i 




i 

X 

i 


2. Implemen- 
tation 

Problems 


X 


X 


1 




X 


X 


3. Overall 
Perceptions 


X 


X 






X 



Design Instrumentation"*- 



Survey 
(Descrip- 
tive Tech- 
niques 



Teacher Quesxion- 
naire(s) 

Student Question- 
naire 
Panel 
Reviews 

On-Site Obser- 
vations 



The table does not- include minor sources of revision data as explained in 
the text. 

An X in a column indicates that the data is collected from a particular 
source or the timing of when the data is collected, respectively • 

Applies only to the h Modules tested in the Spring of I97U. 
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2. Interljn Summatlve Evaliiatlon 

a. Purpose 

The purpose of the interim summatlve evaluation was to collect 
data regarding the impact of the modules on students and to 
provide an additional source of data input into the revision 
process. Information concerning the amount and type of 
student gain on cognitive and affective measicpea was sought. 

b. General Design and Sources of Ihfonnation 

The basic design* of the evaluation can be generically 
diagrammed as: 

R (>L Xi ©2 Experimental Grotq^ 

R ©2 Oj^ Control Group 
Wherein 0^ and O3 represent pretests, O2 and represent post- 
tests > Xi stands for the experimental treatment (i»e., the 
module) and R equals random assignment of individtxals to treatment 
group. Stated differently, the design consists of comparable 
experimental and control groups, where the experimental groi?) 
received the module and the control group did not. Both groups 
would receive two (2) instruments as a pretest and the same two 
as a posttest. The instruments consisted of a multiple choice 
type knowledge test and a brief attitude scale with some 
additional open-ended questions. The questionnaires and data 
sources described earlier for formative evaluation also 
contain opinion type questions which will help to determine the 
impact of the modules. 



slight variant of this design was used for the second set of h modules tested 
in the Spring of 1974. 
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Data collected from the two groups will be presented in 
tables iJi the module specific sections of the notebix>lc. For the 
most part, the tables will be descriptive in nature. Some of 
the data, however, will be anetlyzed through a rather complex 
statistical procedure. To assist both the reviser and the 
reader of the notebook, data tables along vith brief interpretations 
of the data will be included in the laodule specific section. 

Table V is a descriptive overview of the Interim Sumnative 
Evaluation Portion of the Pilot Test. 
Sampling 

1. General Sampling Framework 

Ideally, the sampling process would be based on a randomization or 
a randomized procedure. By means of randomization (either assign- 
ment or selection) it would be possible to say that both the control 
and experimental groups were, within the laws of chance, equal. 
That is, the groups voxild essentially be equalized in terms of 
I.Q,, reading ability, prior knowledge of careers, etc. Further - 
more, if students were randomly selected and/or assigned to control 
and experimental groups from a well defined population it would be 
possible and legitimate to generalise the results of the evaluation 
back to that population. 

Given the above framework, the difficulties of S6a?)ling in this 
situation become apparent. First, to work in the schoolfi with 
modules of the type produced in this project requires the 
cooperation of many school personnel. Small groups making 
extensive use of media will in many instances change or alter 
normal school oreratlon and regular classrocm routine. As a 
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result y it may have been harder to locate schools willing to 
paurticipate in the pilot test of the modules. Hhus the schools 
and the teachers who participated in the pilot test of a module 
were volunteers. Volunteers may differ ftom the so called average 
school or teacher (whatever those undefined entities ml^t be). 
Secondly y most schools cannot easily handle the random assignment 
of students to control and experimental groups • hence » most of 
the time^ students were volunteered from the context of an intact 
classroom* (Note: the teacher questionnaires contained several 
questions regarding how students were selected for a module.) 

Several other factors weighed heavily in the saiq)ling procedure. 
First 9 the pilot test of a nodule was seen as an individual ..xperi- 
ment. The systematic study of the impact of more than one module on 
a student was not only not undertaken, it was consciously avoided due 
to the early developmental nature of materials and time constraints. 
Hiis made it necessary to involve a good number of both Jeffco 
and Denver junior high schools due to the fact that fresh, uncon- 
taminated samples were needed for testing each module. Second, the 
Denver Public School System situation is unusual in the Denver is 
presently busing limited numbers ^ f students for the purpose of racial- 
ly balancing the schools. Hie cost of collecting data to describe each 
school ptpulation involved in the pilot test did not seem warranted, 
so in the module specific section only very brief description of the 
particular schrol will be given. This will be a qualitative 
description developed by an administrator (or administrators) familiar 
with the school in question* 
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2. Types of Claases to be Involved 

As noted earlier the modules were not designed with a specific 
sxibject area in mind. It was felt that they cotild flexibly be 
used in a variety of subject matters areas such as English, social 
studies, math«na;ics, etc. The materials offer opportunities for 
educators in different areas to incorporate career education into 
their respective curricxila. The modules could also be used with 
volunteers from clubs, study halls, etc. 

For some modules, however, specialized equipment, settings- 
and teachers w^^: required. Where this is the case, the text 
of the module specific section will include reference to the 
Ijjniting nature of the module. 

When students were selected to participate in a module 
from intact classrooms, em atteiq>t ^ms made to choose control 
grouDS from claasrooms similar, to those of the experimental 
groups. For exaiz^ple, if an English class provided the experinental 
group students, then another English class was designated as a 
control group, where possible. If the latter arrangement could 
not be established then, perhaps, a social studies classroom 
was selected. 

3. Types of Grouping 

Five types of experimental groxipings of students vere anticipated 
as the pilot test was started in the Denver and Jeffco schools* 
These groupings irere a means of accommodating the program within 
the constraints of existing buildings and programs. Generally, 
they range on a continuum from most desirable to least desirable 
from the standpoint of the original intent of the Occtqaational 
Exploration Program* They are: 
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^ ^Staall group pxxlled out of regular classroom; moat 
desirable in that the simulation could take place 
free of competing influences; 

» -Small fproups in regular classrooms; not as desirable as 
the above case in that the teacher would have to 
simultaneously coordinate the efforts of two groups 
of students and develop some sort of alternative 
activity for those students not participating 
in the module. Also, there would most likely be 
snace and noise problems; 

- -Two experimental groups in regular classrooms ; less 
desirable than the previous two groupings in that group 
competition is introduced as well as all the problems 
specified in the grouping described directly above; 

—Whole class involved (doubled roles) ; given the 
limited number of materials supplied with a laodule 
this method of grouping is simllor in desirability to 
the grouping described immediately above« The press 
on materials could lead to serious problems In 
implementing a module* 

- -Students selected and grouped other than by use of 
intact classrooms ; in this Instance, students would be 
selected (or would volunteer) ftrom a study hall, an 
after school club, a gym class, etc* Although this 
grouping is listed last, it most closely 
parallels the first grouping listed above. 

The importance of grouping may have considerable bearing on 
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hoir the module was received by students and teachers* The 
reviser should he alerted to and aware of the type of grouping 
as he analyzes the revision information , particularly for a 
grouping that may have had a negative impact on the module, 
Ihstruroentation 

A. Knowledge Test (Wl;at do You Know? ) 

1. General Structure - aj described earlier, the knowledge tests 
used per module wer^* structured arotrnd the objectives found on 
page 8 • Each module will differentially deliver on those 
objectives, hence the test for one modxile might be more heavily 
focussed on knowledge of respcnsibilitles, whereas that for another 
might cTiphasize knowledg'^ of processes* The determination of imat 
to test for was based upon the Judgzu^nt of the project evaluator 
and other project staff involved in developing the module* The 
focus was generally limited to only two or three types of knowledge 
due to the amount of time etLlowed for testing students* 
(See point #3 later in thta section. ) 

!nie reviser shoiild note that the modixle may deliver knowledge 
beyond that tested for with the What Do irou Know? instrument. The 
items in the test represent only a two time sampling of a limited 
spectnun of achievement* Attention must be paid to the other 
sources of data from the pilot test with regard to additional as 
well as serendipitous types of learning/achievement. 

2* Types of Question s 

Easically the ^ests consist of multiple choice questions or 
'derivatives thereof. For example, the tests contain questions 
which require selecting the correct answer from one of four 
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choices or determining whether a statement is true or false 
(selecting the correct choice from one of two options). There 
are some matching type questions in addition to the multiple 
choice types. 

The content and diffic\aty does vary somewhat. First, an 
attempt was made to focus on concepts presented in the modules 
that related to understanding careers (e.g., what are the ;Job 
responsibilities of various occupations within a field )• Minutia or 
trivial detail was avoided in question content as much as possible. 
Secondly, some of the questions were aimed at a somewhat higher 
level skill than knowledge. Each test had at least several 
aoplication or problem tynes of multiple choice questions. 

An average item difficulty of .50 is the ultimate goal in test 
construction. This would tend to yield the most reliable test 
and help to overcome the limited numbers of items that were 
dictated by constraints (see nert section) in the pilot test. 
Reliability estimates viU be provided in the module specific sections 
to indicate the degree to which the goal was achieved. 
Approximate Length - the tests do vary in length within a set of 
broad constraints. Time, in terms of test administration, was 
crucial in that it was the judgment of project staff that no more 
than one class period (about Ko^k3 minutes) could be allotted to the 
pretesting and posttesting, respectively. Hence the knowledge 
test would have to take quite a bit of time less than 33 
minutes if an affective or attitudinal measure was to be 
administered during the same period. Test reliability would thus 
tend to be decreased. 
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Given this constraint, the tests for the 
first three Modules vere a naxiraum for the first 3 modules of 25 
questions in length. However, sc»ne questions had several 
sub-parts so that the maximum length In terms of questions 
and sub-parts (which are treated as separate questions) was 
U5 questions. For the second k modviles tested in the Spring of 
197^, the tests generally contained about 3U questions. If the 
reviser does decide to study the test result tables in the 
module specific section, it is advised that the test length be 
taken into account when locking at test reliability. 
Scoring Procedures 

Each question (and sub-part) is scored as thoijgh it was a single 
question of equal weight. No differential weighing of question 
will be applied at this time. Due to limited lead time in pre- 
paring tests for pilot testing of the first three modules, it was 
not possible to incorporate answer sheets into the test 
administration sessions. Thus, the scoring? procedure assumes 
somewhat greater importance than it would in other circumstances. 
First, a code notebook vas prepared, then student answers were 
transferred to the right hand margin of the page in order to 
facilitate kt;y pmching. The transferral of responses was then 
checked and Hollerith data cards were punched and machine 
verified. (Note, for the second U modxxles tested in the Spring 
of 197^», machine scorable answer sheets were utilized^) 

Data scoring and test analysis was then conducted by use of a . 
standard program located in the computer library of the 
Evaluation Center of The Ohio State Ifaiversity* The program 
produced such data as a listing of test scores per individual, 
test means, test standard deviations, reliability coefficients, 
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item indices 9 etc* These will be found in the module specific 
sections of the report* 
B. Affective Test (What Do You Like? ) 

1. General Structure (Objectives) - the What Do You Like? scale vas 
designed to measure or begin to assess the effect of the nodule 
on student attitude. In this case, it is difficult to define 
attitude. Attitude towards whftt? And at vhat level or depth of 
the attitudinal domain should the instrument be aimed? In addition, 
the constraint of time (perhaps 10-12 minutes of class time for 
test administration) greatly precludes the use of a lengthy in«- 
strument. 

A decision was made to focus the attitudinal scale on one aspect 
of affective behavior. That is, what are the student *s preferences 
for either functions or types of ottupations presented in the module. 
Would a student as a result of paurticipating in a module, have a 
stronger, more defined preference for an occupation? Could a student 
justify or better explain his reasons for having a preference from 
taking the m dule? The attitude scale was seen as a preliminary 
start in the direction of measuring a student's beginnixig awareness 
of occupations. To the end of measuring awareness, a brief scale 
{5^10 questions) was constructed and administered to experimental 
gr :>up students. Besides assessing awareness several open«-ended 
questions dealing with selecting or exploring occupational possibilities 
were included in the instrument. Tliese dealt with a student's 
perceptions of what types of experiences a person should have 
before he/she enters the world of work and what an individual 
should take into consideration before selecting a job* 
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Type of Questions - TBiere were several basic types of questions 
included in the What So You Like? instrument. The first type 
contained two parts - one dealing with a preference for an occu- 
pation or occupational function and the student's reasons for ex- 
pressing that preference. The student initially responded to the 
preference scale and then was given space to state reasons for his 
choice. For sca» of the attitude scales a second page was added 
which atllowed the student to supply his own occupations or functions, 
to state his preference for them, axkl to then state his reasons for 
that preference. 

The preference scale consisted of the following four scale 
values : 

—Yes, I would like to try this (occupation or function); 

--I'm uncertain about trying this (occupation or function); 

—No, I would not like to try this (occupation or function); 

—I do not have enough Inforaation to make a decision. 
It was felt that as a result of participating in the module students 
w uld develop stronger preferences, and ones that they could more 
definitively support or explain. In accordance with that perspective, 
the scale was scored so that the stronger the preference the 
higher the score. The details of the scoring system will be given 
in point four below. 

In addition to the preference questions, three-four open-ended 
questions regarding how one mi^t go about preparing for an occupation 
or selecting an occupation were included in the What Do You Like? in- 
strument. For the four modules tested in the Spring of 1971*, only one 
open-ended question was used. (This question required the students 
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to complete a conversation regarding Jobs in the particxilar 

field being simulated. Again, it was felt that as a result 

of nodxile participation the students wo\ild be able to have a better 

idea of the job select ion/preparation process.) 

Approximate Length 

As in the case of the knowledge test, length was greatly con- 
strained by time factors. One class period was the total time 
available for both the pretesting or posttesting, respectively* 
Given the length of the knowledge test (35-^5 questions and 
question pecrts, 25*35 minutes) the attitudinal measure had to be 
fairly brief. In general, 10-15 minutes were allotted for 
administering the instrunwnt. The scale, therefore, was about 
2-3 pages long with about 8-15 questions for the first 3 modules 
and 6-7 questions for the second set of k modules tested. 
Scoring Procedures 

Scoring of the What Do You Like? scale is somewhat complex. For 
the preference questions, the stronger the preference the higher 
the score. In other words, if the student stated that he was or 
was not interested in a career, he received a score value of three. 
If he was uncertain of his preference he received a score of 2 
and if he possessed insufficient information on which to make a 
choice he received the lowest possible value of one. The number 
of reasons was then tallied and for each reason a score was given 
based upon the degree to which the reason supported or fitted the 
preference choice* Reasons were also coded as to type, i.e., 
financial, ability, enjoyment, etc. 
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Other open-ended questions were scored according to number of 
reasons siipplied, the degree to which each reason fit the 
question^ and type of ret son. In no instance were more than three 
reasons coaed. This is bcsed on the observation that more than 
three reasons being given was a fairly rare occxirrence. 

The strategy for scoring responses and categorizing reasons 
was developed after a preliminary analysis of data. Randomly 
selected forms were studied to see if the responses (reasons or 
explanations) given by students fell into categories. From this 
tyre of analysis a scoring scheme and procedure was established. 

During the scoring of forms considerable judgment on the part of 
the scorer/rater must be employed. Detailed instructions for scoring 
were written and adhered to as much as possible • Records of inter- 
rater reliability were maintained. One person rated the forms 
and transferred responses to IBM machine scorable ans\^er sheets. 
A second rater rated a sample of the attitude scales and the degree 
of consistency among the raters was established. 
Student Post I^odule Questionnaires (What Do You Think?) 
1. Cteneral Structure 

I'he post nodule questionnaire is quite different from the knowledge 
and attitudinal instruments. The latter two are closely related 
to the interim summative evaluation that was described earlier. 
As noted previously, these instruments have somewhat limited 
utility in regard to formative or revision types of concerns. 
The post module questionnaire, on the other hand, is designed to 
assess or to collect information about, the students* perceptions 
of the module • For exaii?)le; What did the student like most about 
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the module?, What did he like least about it?, What types 
of problems did the students find in the modules?, etc. 

Students were encouraged to respond freely. The questionnaire 
was considered to provide some valuable information for the 
revision process. 
T^es of Questions 

The questionnaire consists of a veuriety of question types. 
The first twenty questions are of a short attitude scale 
requiring approximately 5-10 minutes of the student's time. 
Beyond this there are a series of short questions which genercdly 
require the student to indicate his perceptions about a topic by 
circling a choice in a scale. Space was also provided for students 
to supply comments emd suggestions. (For the second h modules, 
the instrument was slightly modified.) 
Approximate Leng th 

The module questionnaire was administered sometime (usually 3 
to f) days) after the completion of the module and the posttesting. 
Since the questionnaire related to perceptions of the module, it 
was administered to only the experimental group. The attitude 
scale in the questionnaire was 20 questions in length followed 
by 10-15 open-ended questions. Total length therefore was 30 
plus questions and the average administration time was about 20- 
25 minutes or more. The questionnaire was free of the competing 
influence of other instrtiments in that it was administered by 
itself. 

Scoring Procedures 

As in the case of the KVhat Do You Like? scale, scoring of the post 
module questir>nnaire for students is fairly complex. The first 
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twenty questions are un attitude scale in which the mast positive 
response per question was given the highest score value. And 
conversely the most negative response per question received the 
lowest score value. The response to these twenty questions could 
be summed and treated as one scale or they could be divided into 
subsets of questions and subscores could be totaled. Since the 
scale was designed to measure or assess feelings about several 
different attributes of the module it is most probable that the 
latter course of action will be followed. (Note: The number of 
items per subset will be small and reliability estimates will not 
be computed inasmuch as they will be greatly limited in utility^ ) 

For the remainder of the questions in the What Do You Think? 
questionnaire the scoring was as follows: 

1, If the question contained a scale, the scale value of the 
students response was simply recorded; 

2« For questions requiring open-ended responses, the 
responses were collated with attention paid to ftre- 
quently repeated types of responses. 

Concerns regarding the quality of the questionnaire data may 
arise as one begins to get involved in the revision process.. Since 
the questionnaire covers many topics it is difficult to obtain 
(and/or develop) measures of reliability and validity. This problem 
occurs not only for the student post module questionnaire but 
for teacher panels and questionnaires as well* Each module speci- 
fic section contains data and/or discussion regarding instrument/ 
data collection method, reliability, and validity. However, of 
special importance to the reviser is the idea of information source 
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convergence or corroboration. While it nay be difficult to as- 
certain if an individual piece of data is reliable/valid, if 
two-three independent or someirhat Independent sources converge 
on the same point with the same information then the set of 
data is quite powerful , Strength emerges ftrom the set rather 
than the individual data source. In other words, the trend of the 
data is a key point to consider in revising a module. 

Teacher Evaluation Log 

1, General Structure 

The Teacher Evaluation Log differed slightly for the first two 
modules (Education and Construction) pilot tested and the third 
Manufacturing), Trom the standpoint of simplicity it is perw 
haps easiest to explain the Log for the third and then to 
briefly describe the difference ftrom the first two. The Log 
is 6 separate instruments \mder one cover. The first five instru- 
ments parallel the parts of a module. That is, the first deals 
with the Introduction to Simulation booklet and sound -on -slide 
presentation that students ste before starting a module « The 
next instrument contains questions about the module preview, the 
next one about the preparation phase, and so on« The last or 
sixth instnsnent relates to the teacher* s overall perceptions 
about the module. 

The Log therefore was a device for teachers to record their 
comments as well as their answers to specifi . questions as they 
worked with and observed students progressing through a raodtile. 
The Log differed in the case of the first two modules in that the 
last Instrument, the General Module Evaluation for teachers, was 
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separated from the ether five instruments. The results from the 
"og may have been affected due tc the fact that some teachers 
felt the quality of the last instrument was somewhat better than 
the first five. 

Teachers were not remunerated for the time necessary to 
complete evaluation instrxanents, (Note: they were paid for the 
time necessary to train them for using the iiodules and for partici- 
pating in the post mod\ile panel reviews.) This coupled with the 
length of the Log led to spotty completion of the forms. The 
reviser should be aware of this occiarrence as he works vith the 
revision data. 

For the h modules tested in the Spring of 197^> the Log 
was deleted from use and a mid-module and post module ' teacher 
questionnaire were substituted i \ its stead. In addition, 
optional forms - a Media Checklist and a Daily Inventory of 
Perceptions (DIP) were made available to teachers. These last 
two forms allow teachers to comment specifically on the im»dia used 
and to provide project staff with any other opinions they might 
have, respectively. 
Types of Questions 

Question form or type varied considerably. Three basic types of 
questions will be found in the Log and in the teacher quest '.onnaires^ 
They are: 

1. Scaled cfuestions with space for open-ended' comments. 
Teachers responded to questions by indicating their 
preferences on a scale ♦ If they had additional commenv.s 
they were encourated to place them In the space so 
provided. ; 
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2. (^pen-ended questions with anrple space provided for responses.; 

3. Questions requiring the teacher to fill, in or supply some 
limited information e.g., numbers of students 
participating, types of students that experienced difficulty 
with the module, time spent on a particular task, etc. 

The instruments in the Log used by teachers to monitor or spot 
problems as they occurred during the participation phase of the 
module was similar in form to the Lesson Evaluation Form developed 
for use in the Comprehensive Caxeer Education Project, the school 
hosed model. (This applies only for the first 3 modules tested. For 
the last h modxales tested, the mid and post module questionnaires were 
used in place of the logs.) This particular questionnaire if ex- 
tensively filled out by teachers axKi if extensively corroborated by 
other data sources will be a most valuable input into the module 
revision process. 
Approximate Length 

To begin, the length of the Teacher Evaluation Log, in terms of 
both teacher time and pages, is dependent on the length and 
complexity of the module* It was asmmed that during the entire 
life span (3 1/2 6 weeks) of a nodule the Log vovld take a 
minimum of about 2 l/2 hours to cooqplete; at a maximum it vould 
expand to nearly 3 hours. As noted earlier the Log atten^ts to 
elicit open<»ended responses from teachers and hence a great deal 
of paper space is required* Checkllsts^vhile possible, would have 
necessitated some preliminaxy observations of module classroom 
behavior before the instruments could have been meaningfully 
constructed. Secondly, the time frame for instrument devel- 
opment would have gone beyond that available. So the LOG^s 
are large in appearance which may, in tum^ have affected the 
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extent to which they were completed by participating teachera. 

The two questionnairea (and the ortional forma) uaed for 
testing k modulea in the Spring of 197U were qxiite a bit ahorter 
than the WG. The two queationnairea required approximately 
25-30 minutea apiece to complete* 
Scoring Procedxirea 

Since only a maximum of U teachers were included in the 
experimental groups, the acoring of the LOG'S is fairly simple* 
Scaled aa well as open«»ended responses will be suwed and 
collated respectively* Ifo detailed scoring is anticipated at 
thia time, (Questionnaires used for the k modules tested 
in the Spring of 1974 will be scored in a manner similar to that 
described for the lOG'S above.) 

Panel Reviews Procedure 

1* General Structure 

Bie panel reviews were designed to collect in-depth Information 
for revision. After completion of a module, teachers were convened 
in a panel setting to discuss, at length, each part of a module* 
They were informed of the panel review session prior to their 
starting of a module* For the sessions they were asked to bring 
in their l£)GS, questionnaires and/or any other notes th^ might 
have made. 

The panel session was not simply for the pmrpose of duplicating 
the results obtained from the LOG or questionnaires but specifically it 
was to probe for more in-depth opinions of teachers. !Bie pcuiel leader (s) 
was (were) responsible for generating a written report summarizing 
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the comments of teachers. It should also be noted that in the 
session^ the teachers vere given the opportunity to express their 
feelings as to how the module should be revised. This type of 
input from classroom practitioners is frequently invaluable for 
revision purposes. 

For the h modules tested in the Spring of 197^ two panel reviews 
involving teachers were conducted. The first occurred midway 
through the module and the second, at the end of the module. The 
general structure described above was employed for both panel reviews. 
Types of Questions 

Basically, 6nly h questions were used for the pemel review. 
They are: 

- What do you feel were the strengths or strong points in this 
part of the module?; 

- What do you feel were the weaknesses of this part of the module?; 

- If you did try to overcome the weakness, what specifically did 
you do and how well did it work?; 

- What recommendations for change would you make in this part of 
the module? 

The panel leader was instructed to probe into these questions as 
teachers responded. For example, why was this a strength, how did 
your students respond, why was this a weakness, '^Id this part require 
too much time, etc. The panel leader probed also for the degree of 
agreement amongst the teachers. In addition to their perception of 
module parts, teachers were asked about their feelizigs to^taro "i^he 
entire module. Notes were kept by either the panel leader or an 
observer if one was present. 
Approximate Length 

Length in this case is dependent on many factors - length of the 



module, module complexity, module problems, interest of the teachers, etc. 
As a general rule, at least 2 l/2 hours was required when a single panel 
was conducted for a module* (When two were conducted somewhat more 
than one hour was required per session,) Frequently, for the 
single panel case, more theui one session was needed. 
Scoring Procedures 

Fox revision purposes, the panel review report will sinqply be 
summarised and included in the Reviser *s Information Stumnary^ 
F, ( bserver Form (Modules tested in the Spring of 197**? only) 

1. General Structure 

For the k modules tested in the Spring of 197U, classroom observation 
wa^ also utilized as a means of collecting data. At each school 
involved in the Spring testing, an observer was stationed to provide 
em independent source of information regardixig how each modiile wu 
being implemented. A very brief three page observation form was 
provided for the observers to complete as they watched students 
and teachers. It was felt that data of this type would be a valuable 
source of input into the revision process. 

To facilitate both the observation process and the use of audio- 
visual devices the four simulations were conducted in a special room 
aesignated for that purpose at each school. The situation was 
somewhat akin to a laboratory setting. 

2. Type of Questions 

There are three basic typjes of questions included in the Observer 
Form. The first is concerned with the use of booklets and media during 
the class period. The second involves comments on obsei'vations the 
observ»3r has made during the period but which do not seem to fit into any 
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category on the form. The third major category relates to group 
or individual activities in the simulation as well as to the 
interaction between students €uid teachers. This type of information 
will serve as an anchor for other types collected in the evaluation. 

3- Approximate Length 

The form is short (3 pages) end designed to be used relatively easily in 
the classroom by the observer. Directions as well as written exan?>les 
were provided to facilitate completion of the observer forms. 

^. Scoring Procedures 

To the extent feasible the data collected from the observers will be 
conipared and if possible collated. To a degree, this was a trial nin 
at this method of obtaining information. CoiH)arative study may indicate 
that it cannot be collated and that for future project use the forms 
and the procedure may have to become more standard in nature. 

SUMM/VRY - Further Considerations 

As the reader will recall, the front section of the evaluation 
notebook was designed to give prospective users of the notebook an 
overview of both the Occupational Exploration Project and the evaluation 
that was conducted during the project yeeir 1973- 197U. Due to the 
heavy revision/formative focus of the first year evaluation, the text 
of the front matter contains frequent references to the revision 
process. It is hoped that this will help to facilitate the reviser's 
job. 

The succeeding sections of the notebook are organized into module 
specific sections. These start out with a brief introduction or 
description of the module that was evaluated, followed by the nature 
of the actual evaluation that was carred out* Data tables and 
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interpretation come directly after the textual discussion. Last, 
but of greatest in5)ortance to the reviser, is the Reviser's Information 
Gummary (RIS). This is the summary of the evaluation Information 
as it relates to revision. The reviser should read both the 
introductory text and the RIS in the module specific section first. 
If he/she desires more detailed information, the data tables may be 
referred to. 

It should be e.'uphasized as a closing note that the information in 
the RIS does not constitute a prescription or a mandatory set of 
rules for the reviser to follow. Ihe RIS is a summar ^^ of the 
information collected and interpretation of trends and their potential 
meeming for revision. The evcuLuation data collected this yecur could 
not begin to anticipate nor answer all revision questions. Only a 
limited subset of the questions could be answered in this evaluation. 
Hence, the judgment of the reviser will still play a major role in 
the revision process. That judgment, however, should be ten?)ered 
against curriculum criteria and the revision plan or strategy should 
contain a clear rationale as to why changes were undertaken. 
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